Take Fig. S.1 as an example.

Humans have the remarkable ability to pick up unfamiliar objects and quickly understand their surface properties, such as friction, and dynamics. This knowledge enables us not only to reorient objects using our arms but also to manipulate them within our hands, extending our capabilities beyond what is typically seen in traditional robotics. In this paper, we introduce a custom parallel gripper equipped with commercial 6-degree-of-freedom (DoF) force-torque (F/T) sensors and custom relative velocity sensors (see Figure 1), for in-hand slip-aware control that relies solely on in-hand sensing. The ability to independently measure force and planar velocity introduces new opportunities for intricate robotic manipulation. This hardware combination enables rapid estimation of friction and contact surface properties without the need for external sensors, thus facilitating for precise in-hand manipulation of objects in both rotational and translational movements. Slip-aware control significantly enhances the functionality of robotic manipulators by enabling the object-end-effector relative pose to adapt during grasping, thereby extending the operational workspace. This adaptability is particularly valuable in constrained environments, where the manipulator's movement is limited, or for intelligent human-robot interaction, enabling for instance more intuitive and safe handovers. Furthermore, in-hand slippage control opens up new opportunities for multi-arm manipulation of single objects, allowing for the repositioning of grasps without releasing the object, thereby enabling more efficient and flexible handling of larger items. Our system has been rigorously tested across a wide range of experiments, demonstrating its effectiveness and versatility.

The incorporation of appropriate inductive bias plays a critical role in learning dynamics from data. A growing body of work has been exploring ways to enforce energy conservation in the learned dynamics by incorporating Lagrangian or Hamiltonian dynamics into the design of the neural network architecture. However, these existing approaches are based on differential equations, which does not allow discontinuity in the states, and thereby limits the class of systems one can learn. Real systems, such as legged robots and robotic manipulators, involve contacts and collisions, which introduce discontinuities in the states. In this paper, we introduce a differentiable contact model, which can capture contact mechanics, both frictionless and frictional, as well as both elastic and inelastic. This model can also accommodate inequality constraints, such as limits on the joint angles. The proposed contact model extends the scope of Lagrangian and Hamiltonian neural networks by allowing simultaneous learning of contact properties and system properties. We demonstrate this framework on a series of challenging 2D and 3D physical systems with different coefficients of restitution and friction.